Add scalar function performance baselines - #9136
Conversation
Merging this PR will improve performance by 79%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | div_i64_nonnull |
2.4 ms | 1.2 ms | +96.92% |
| ⚡ | Simulation | add_decimal_i128_nullable |
7.4 ms | 3.8 ms | +96.75% |
| ⚡ | Simulation | mul_i64_nonnull |
2.1 ms | 1.1 ms | +96.57% |
| ⚡ | Simulation | add_i64_nonnull |
1,794.2 µs | 915.9 µs | +95.9% |
| ⚡ | Simulation | add_decimal_i64_nonnull |
3 ms | 1.5 ms | +95.07% |
| ⚡ | Simulation | sub_i64_constant |
1,237 µs | 639 µs | +93.57% |
| ⚡ | Simulation | mul_u32_nonnull |
1,391.4 µs | 722.4 µs | +92.6% |
| ⚡ | Simulation | add_i64_nullable |
1,859.1 µs | 967.8 µs | +92.09% |
| ⚡ | Simulation | lt_i64_nullable |
1,242.7 µs | 653.4 µs | +90.18% |
| ⚡ | Simulation | mul_i32_nonnull |
965.6 µs | 509.5 µs | +89.54% |
| ⚡ | Simulation | mul_u16_nonnull |
948.8 µs | 502 µs | +89.01% |
| ⚡ | Simulation | mul_u8_nonnull |
756 µs | 405.3 µs | +86.53% |
| ⚡ | Simulation | mul_i32_constant |
671.6 µs | 364.8 µs | +84.11% |
| ⚡ | Simulation | mul_i32_nullable |
1,032.7 µs | 562 µs | +83.75% |
| ⚡ | Simulation | eq_i64_constant |
680.8 µs | 372 µs | +83.02% |
| ⚡ | Simulation | mul_i16_nonnull |
510.5 µs | 281.7 µs | +81.19% |
| ⚡ | Simulation | mul_i8_nonnull |
289 µs | 170.6 µs | +69.38% |
| ⚡ | Simulation | and_bool_nullable |
186.2 µs | 115.4 µs | +61.28% |
| ⚡ | Simulation | fsl_large |
253.2 µs | 211.2 µs | +19.92% |
| ⚡ | Simulation | or_bool_constant |
78.9 µs | 67.4 µs | +17.12% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/scalar-fn-baselines (ca01627) with develop (3d0c124)
1fb231a to
c82b8f3
Compare
c82b8f3 to
71787a7
Compare
71787a7 to
880ddac
Compare
|
This benchmark has a too long runtime |
880ddac to
69edd3a
Compare
f599518 to
9fcb4a7
Compare
1dccaaf to
bf814bb
Compare
bf814bb to
003cc85
Compare
003cc85 to
9755708
Compare
|
Still too long |
|
@joseph-isaacs what is too long? which one? https://github.com/vortex-data/vortex/blob/develop/docs/developer-guide/benchmarking.md has nothing about benchmarks being too long, can you make a PR there? |
|
column_x_column_polygons | N/A | 23.8 ms |
|
@joseph-isaacs this is unhelpful. There are other benchmarks here that are longer too (i.e. 123ms). Is there some limit here that is undocumented? Why is it bad for microbenchmarks to be "long" in the first place if we're still able to get useful info out of it? |
|
They’re not very micro benchmarks with that runtime. I think in aggregate there’s ~30 non compliant benchmarks. This numbers is an arbitrary threshold we keep to stop them from decaying over time |
|
I usually think under 1 ms is a good target unless there is a very good reason to go above this. If we go above this two things happen:
|
…9176) ## Rationale for this change The benchmarking guide asked for "less than 1ms" per iteration without saying whether that was an aspiration or a limit, and without saying which number to measure against it. That gap showed up in review on #9136, where benchmarks landed at 23.8 ms and 123.4 ms per iteration. The review conversation there had no documented rule to point at — as noted on that PR, the guide "has nothing about benchmarks being too long". This documents the rule first, on its own, so it can be cited independently of any tooling that enforces it. ## What changes are included in this PR? Docs only — one section of `docs/developer-guide/benchmarking.md`: - States that **1 ms is the maximum, not a soft target**. - Says what to do when a benchmark does not fit: shrink the input, split it into smaller parameterized cases, or gate it with `#[cfg(not(codspeed))]`. - Names the number to check: the **per-iteration** time CodSpeed reports under `HEAD` for every benchmark a PR adds or changes — not the runtime of the whole benchmark binary, which is what "this benchmark takes too long" usually gets confused with. The heading loses its `~` (`under ~1 ms` → `under 1 ms`), which leaves the existing anchor `#keep-per-iteration-execution-time-under-1-ms` unchanged. A follow-up branch adds CI that reads CodSpeed's report and comments when a new or changed benchmark exceeds this budget. It is deliberately kept out of this PR so the rule can be agreed on its own. ## What APIs are changed? Are there any user-facing changes? No code, no API changes. Documentation only. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn --- _Generated by [Claude Code](https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn)_ Co-authored-by: Claude <noreply@anthropic.com>
9755708 to
57236cb
Compare
|
@joseph-isaacs everything is sub ms now |
57236cb to
210a2c8
Compare
HarukiMoriarty
left a comment
There was a problem hiding this comment.
The geo part micro benchmark looks good to me!
Measured against develop through #9136's own public-path bench, the RowFn port was 7.6-7.7x slower across every arm at 65536 rows (13.5us to 104us, divan fastest and median of 100 samples, best of two ABBA runs, Apple M4 Max). The non-nullable arms regressed as hard as the nullable ones, so this was never about masking or validity. Half of it was a defect worth recording: BytesLen::get called VarBinViewArray::views once per row, which is a host-buffer downcast, a vortex_expect whose panic path the optimizer cannot hoist, a length division and a slice reconstruction. Pointing Varying at the views slice instead took it from 104us to 52us. Bytes::get had the same bug. This is the trap the InputElement::get doc warns about, and the next person to add an element should assume Varying is load-bearing rather than decorative. The other half is not a defect. Develop's views().iter().map(...).collect() vectorizes to better than one row per cycle, while a strided four-byte field read widened to eight does not. The row executor itself is fine: row_checked_add hits 0.21ns per row on the same machine, matching both a specialized control and develop's byte_length, while reading two input columns instead of one. So byte_length is the outlier, and it is the case #9128 already excludes, where a row is not the natural unit of work. Reverting the port leaves Bytes, BytesColumn and BytesLen with no caller, so they go too, along with the two branch-only byte benches. Shout keeps its coverage as a test-local element, since it was the crate's only non-dense-safe input and therefore the only in-crate exercise of the Filter contract and of branch-and-skip agreeing with filter-and-scatter. The two compile_fail doctests move to inline elements; both were verified against a positive control, which caught two bugs in the rewrite where they had been failing for unrelated reasons. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
210a2c8 to
f0d133b
Compare
There was a problem hiding this comment.
this add 8 benchmarks, I am not sure we need this.
Also it only benchmarks all inline or all not inline
There was a problem hiding this comment.
Im happy to remove this, though we should decide if we want to rewrite a scalar function like this with rowfn or not. It is possible, and it is also possible to get close to parity, but maybe not worth it.
| let validity = Validity::from_iter((0..len).map(|i| i.is_multiple_of(10))); | ||
| let input = MaskedArray::try_new(long_strings(len), validity) | ||
| .unwrap() | ||
| .into_array(); | ||
| bench_byte_length(bencher, input); | ||
| } |
There was a problem hiding this comment.
measures masking and str len
There was a problem hiding this comment.
I don't think we want a like expr and we don't want row fn like lifting
There was a problem hiding this comment.
wait why not? this seems like a good place to do it since it is localized compute with one setup (compile dfa/regex)?
There was a problem hiding this comment.
cause i want to run the dfa over the whole string batch at once
There was a problem hiding this comment.
I removed it just for simplification, but when you say "whole string batch at once" you still have to do it sequentially. And ideally the rowfn machinery should get optimized away so that it looks like its the same
| fn execute(distance: VortexResult<impl IntoArray>, ctx: &mut ExecutionCtx) -> ArrayRef { | ||
| distance | ||
| .unwrap() | ||
| .into_array() |
There was a problem hiding this comment.
do we want this in the benchmark
There was a problem hiding this comment.
Yes I think we want this
There was a problem hiding this comment.
the into_array. I guess that is just a Arc::new()
| .bench_values(|(array, mut ctx)| { | ||
| array | ||
| .into_array() | ||
| .execute::<ExtensionArray>(&mut ctx) |
There was a problem hiding this comment.
does this do a full canonnical?
There was a problem hiding this comment.
whoops
we really should change this
There was a problem hiding this comment.
oh wait no it does do stuff because the input is not an extension array, it is a normalized array
but we should change the extension array execute regardless...
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
|
whoops I deleted like.rs, will bring it back without modification |
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Summary
Tracking Issue: #9128
Adds stable scalar function performance baselines for
byte_length,l2_norm,normalized,inner_product,cosine_similarity,GeoContains, andGeoIntersects.The benchmark sizes are all comfortably above the roughly 146-instruction Divan harness floor from #9011. Each benchmark binary also uses vendored
mimalloc, since scalar function execution allocates its output inside the timed trace and glibc differences between runner images caused the flakes fixed in #8861.Why
Codspeed can only compare a later implementation change when the same benchmark name already exists on
develop, so these baselines need to land before theRowFnthings I want to make. And these are pretty cheap, might as well add them.